Original Article

Comparative Analysis Between Pre-test/Post-test Model and Post-test-only Model in Achieving the Learning Outcomes

Tayyaba Gul Malik, Rabail Alam

Pak J Ophthalmol 2019, Vol. 35, No. 1

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

See end of article for authors affiliations …..……………………….. Correspondence to: Tayyaba Gul Malik Professor of Ophthalmology Rashid Latif Medical College Email: tayyabam@yahoo.com

Purpose: To compare the effectiveness of pre-test/post-test model with post-test-only model in achieving the learning outcomes of a lecture of 45 minutes. Study Design: Quasi experimental study. Study Place and Study Period: Rashid Latif Medical College, in 2018. Sample Collection: Non-probability convenience sampling. Material and Methods: 131 students of fourth year MBBS in a private medical college of Pakistan were selected for the study. Students of the same year and same college were included in the study. Two teaching models were compared on the same set of students at different time-periods. Pre-test followed by post-test model was compared with post-test-only model. In the pre-test/post-test design, the students were given a test before the lecture was delivered. The same test was given after a lecture of 45 minutes. The same group of students were taught another topic in another lecture of 45 minutes. There was no pre-test this time. 50 percent score was selected as the passing criteria. Results: There were 131 students, 66 males and 65 females (ratio of 1.01:1). In the pre-test/post-test model, 82% (n = 107) students passed the test while 5% (n = 6) failed. 14% (n = 18) students scored borderline marks. In the post-test-only model, 57% (n = 74) students passed the test while 6% (n = 8) failed. 37% (n = 49) students scored borderline marks. Conclusion: Pre-test/post-test model was significantly more effective in students in achieving the learning outcomes in a lecture as compared to post-test only model (p < 0.01). Key Words: Pre-test Post-test teaching model, Post-test-only design, Evaluation.

Evaluation in teaching is an integral part of successful and effective teaching¹. It is defined as "the process of obtaining information about a course or a program of teaching for subsequent judgment and decision-making"². The importance of evaluation in medical education cannot be overlooked, as evaluation drives curriculum. A sound curriculum in medical education breeds good clinicians, which improves the healthcare of the patients. Hence, good medical teaching is directly concerned with development of good clinicians^3,4,5. Pre-test/post-test and post-test-only designs are important assessment tools that help in direct and effective evaluation of a course or lecture to improve student learning. The idea of pre-test/post-test evaluation model is to measure baseline knowledge of participants at the beginning of a course/lecture and compare it with the knowledge gained after the course. Comparing participants’ post-test scores to their pre-test scores enables to see whether the activity was successful in

Fig. 1: (Above) showing the pre-test/post-test design. (Below) Post-test only design is shown.

increasing participants’ knowledge of the taught content. In the post-test only model, the design is the same as pre-test/post-test but the pre-test is omitted. The idea is shown in its simplified form in figure 1.

“Outcomes of teaching” is a broad term, which encompasses not only the acquisition of knowledge but also practical skills and attitudes^6,7. In lectures, skills and attitudes cannot be assessed. Pres-test/post-test and post-test only models are just one aspect. It must be combined with other assessments, for example, peer evaluation and program review to present an authentic and holistic data to reflect the educational gains³.

In this study, we have tried to find out which of the two evaluation designs; pre-test/post-test model and post-test-only model is more effective in achieving the learning outcomes after 45 minutes lecture in a class of fourth year MBBS, in a private medical institution of Punjab. To the best of our knowledge, this is the first comparative study between pre-test/ post-test and post-test-only designs.

MATERIAL AND METHODS

131 students of fourth year MBBS in a private medical college of Pakistan were selected for the study. The study was approved by the Institutional Review Board. The inclusion criteria was students of fourth year MBBS, age between 21 and 23 years (average 22 years), irrespective of gender. Students of the same year and same college were included in the study. Students from other MBBS classes and other colleges were not included.

Pre-test/post-test model and post-test-only model were compared on the same set of students at different time-periods. The lectures in both models were delivered on two different topics of Ophthalmology. Pre-test followed by post-test model was compared with post-test-only model. In the pre-test/post-test design, the students were given a test before the lecture was delivered. The same test was given after a lecture of 45 minutes. The questions given in the test were problem-based and not just the recall of knowledge. The pre-test, post-test and the lecture were done on the same day. The same group of students were taught another topic in another lecture of 45 minutes. There was no pre-test this time. However, post-test was given on the same day immediately after the lecture was over. In both evaluation models, the lecturer was not changed. The tests were prepared and scoring was also done by the same teacher. 50 percent score was selected as the passing criteria. Scores between 45 and 49 percent were regarded as borderline and less than 45 percent score was considered fail. The data was collected, compiled and then analyzed using chi square test.

Results

There were 131 students, 66 males and 65 females (ratio of 1.01:1). In the pre-test/post-test model, 82% (n = 107) students passed the test while 5% (n = 6) failed. 14% (n = 18) students scored borderline marks. In the post-test-only model, 57% (n = 74) students passed the test while 6% (n = 8) failed. 37% (n = 49) students scored borderline marks. Pre-test/post-test model was significantly more effective in achieving the learning outcomes in a lecture as compared to post-test only model (p < 0.01). According to frequencies and percentages, the pre/post-test model is providing more help to students to pass the test and post-test-only design gives higher frequency of students at borderline (Figure 2).

Fig. 2: Comparison of pre-test/post-test model with post-test-only model.

DISCUSSION

Research in medical education is increasing day by day and advancements in the research methods is gaining momentum. Research methodology in medical education uses techniques which are adopted in psychology, epidemiology and related fields⁸. Different types of research designs, which are used in these fields include “one shot case study”, one group pre-test-post-test, two group pre-test/post-test, post-test-only, two group randomized post-test-only design and Solomon Four-Group Design. In “one shot case study” there is only one group, which is exposed to intervention and there is no control group⁹. In “one group pre-test-post-test design”, there is no control group and the pre-test is compared with the post-test^10,11. “Two group pre-test/post-test design” compares the intervention group with the control¹². Although pre-test/post-test design is more widely used in medical education, some researchers have found “two group randomized post-test-only design” to be more useful provided there are 40 participants in each group¹³. The Solomon Four-Group Design is another model in which there are four groups; two with a pre-test (experimental and control groups) and other two without pre-test (experimental and control groups)¹⁴.

Pre-test-post-test and post-test-only designs are widely used in behavioral research but less frequently employed in medical education research. In annual system of education, the results of the students in final examination is the only way to evaluate the teaching methodology in the previous year. Pre-test/post-test and post-test-only designs provide measurement of change for assessing the impact of teaching during academic year.

There are many situations where a pre-test is either impossible or difficult due to time constraints. In such conditions, post-test-only method can be employed. Many studies are available which have shown that pretest/posttest model helps to monitor student progression and learning throughout a course or program¹⁵. This technique is not only used at many educational setups to test the success of a teaching session but is also found to be a tool of research in medical education.

The results of this particular study showed that pre-test/post-test model is comparatively more effective in achieving the learning outcomes in a lecture setting. By Michael Delucchi, a pretest/ posttest technique, once put into practice, can be used to improve the process of teaching skills¹⁶. For example, the topics and areas in which students show poor performance in post-test can be revised and later given increased emphasis. The drawback is that extra time has to be devoted to these topics. Both the instructor and students can benefit from a pretest/ posttest course design. In this particular research, the tests were taken with hidden students’ identity. However, if the identity of the students is discernable, we are able to know very weak and strong students in the class. Students showing poor performance can be identified for extra coaching. With pre-test, difficult topics are determined and further planning of the lectures is done keeping in view all the difficult points. This is not possible with post-test-only model. This is the reason, why pre-test/post-test design was more effective in achieving the desired learning outcome. In our research, the pre-test and post-test were performed on the same day. If pre-test is given before the start of a course, the lecturer/teacher is able to know which topics to stress upon and which topics to touch lightly based on the response of the learners.

There are some recommendations by T. Wood & G. Cole, regarding use of pretest/posttest technique¹⁷. Firstly, the instructors should not include pre-test/post-test scoring as a part of final assessments. Secondly, questions should be formed based on the primary learning objectives. He further suggested that if the teacher is not able to write a test item on the learning objective, then he/she should rewrite the learning objectives. The question items for pre-and post-tests can be multiple choice, true/false and short answer. They should be created in a clear manner. Faulty questions cannot accurately measure changes in knowledge. A very important proposal is that the questions should not include material, which demands the memorization of minor unimportant details. The students should not be tested on whether they remember a particular term or percentage etc. Rather they should be tested on important concepts and related facts.

To determine the short-term and long-term effects, the post-test should be administered directly at the completion of the course and also at a later date to measure long-term impact of learning. Later assessment was not part of this study. There are few disadvantages of this technique. In very short teaching programs, these tests do not meet the time requirements. Secondly, pre-test is only useful when a student has some relevant baseline knowledge on the course topic. Another objection to good performance in post-test of pre-test/post-test design is that the students taking the test for a second time mostly do better than those taking the test for the first time making the results biased⁷. Some students may drop out of the course before the post-test has been conducted, resulting in post-test results that may be higher than they should be because those who remained in the course were more successful. In our study, the number of students was same in pre-test/post-test design as well as post-test-only design. So, dropout flaw was effectively taken care of.

Boston University has given certain guidelines for developing a pre/post-test¹⁸. They recommend to create 10 to 15 questions that could test students’ knowledge of a learning outcome at the end of a course. If a course has more than one topic, all the faculty members, teaching the topics should meet to create these questions. Difficulty level of the questions should be according to the level of the students. For example, post-graduate students will have questions with different level of difficulty from the under-graduate students. Questions in pre-test and post-test should be exactly the same.

According to Martyn Shuttle worth, pre-test/ posttest model was a derivative of post-test-only model¹⁹. One of the drawbacks of post-test-only design is that we do not have pre-existing knowledge of the student, it cannot be said that the score in the post-test is the outcome of intervention (lecture in this particular case).

Outcome of teaching is not only the increase in knowledge but also improvement of practical skills and development of professional attitudes. Unfortunately, both these models are not effective in evaluation of skills and attitudes. Most of the studies available in literature evaluate a single or two components of a course or program and are not comprehensive²⁰.

In addition to traditional “pre-test before the class” and “post-test after the class” design, there is “post-then-pre test design” called “retrospective pre-test/post-test”. In this design, the learner is asked to first report present behaviors in a post-test and then, their perception of the same behavior before taking the course (a pre-test equivalent). Because the student is asked their perception of improved performance in the same reference of the post-test, some educators feel this is a more accurate measurement²¹.

The positive points of our study is that, it made comparisons between the same people, or groups of people, at different points in time. Limitations of this study are that the tests in both models were different. Another drawback was that only short-term effect was studied. No long-term impact on knowledge was seen in this study.

CONCLUSION

Pre-test/post-test design is more effective in achieving teaching goals in a lecture setting than post-test-only design.

Author’s Affiliation

Dr. Tayyaba Gul Malik

Professor of Ophthalmology

Rashid Latif Medical College

Rabail Alam

Assistant Professor

Institute of Molecular Biology and Biotechnology

The University of Lahore

Author’s contribution

Dr. Tayyaba Gul Malik

Research planning, data acquisition and analysis, literature research, manuscript writing and final review.

Rabail Alam

Statistical analysis, final review of manuscript.

Conflict of Interest: None

REFERENCES

1. Snell L, Tallett S, Haist S et al. A review of the evaluation of clinical teaching: new perspectives and challenges. Med Educ. 2000; 34 (10): 862-70.

2. Goldie FJ. AMEE Guide No. 29: Evaluating educational programmes. Med Teach. 2006; 28: 210-24.

3. Prideaux D, Alexander H, Bower A et al. Clinical teaching: maintaining an educational role for doctors in the new health care environment. Med Educ. 2000; 34 (10): 820-6.

4. Cook, D. A. Twelve tips for evaluating educational programs. Medical Teacher, 2010; 32 (4): 295-301.

5. Gilman SC, Cullen RJ, Leist JC, Craft CA. Domains-based outcomes assessment of continuing medical education: the VA model. Acad Med. 2002; 77: 810–7.

6. Blumberg P. Multidimensional outcome considerations in assessing the efficacy of medical educational programs. Teach Learn Med. 2003; 15 (3): 210–214.

7. Frank JR, Danoff D. The Can MEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach. 2007; 29 (7): 642–647.

8. Carney PA, Nierenberg DW, Pipas CF, Brooks WB, Stukel TA, Keller AM. Educational epidemiology: applying population-based design and analytic approaches to study medical education. JAMA. 2004; 292 (9): 1044-1050.

9. Kerlinger, F. N. Foundations of behavioral research (2nd ed.). New York, NY: Holt, Rinehart, and Winston, 1973.

10. Campbell, D. T., & Stanley, J. C. Experimental and quasi-experimental designs for research on teaching. In Handbook of research on teaching. N. L. Gage (Ed.), Chicago, IL: 1963 (pp. 171-246).

11. Knapp TR. Why is the one group pre-test post-test design still used? Clin Nurs Res. 2016; 25 (5): 467-72.

12. Pat D, John T. Analysis of Pre‐test‐Post‐test Control Group Designs in Educational Research. Educational Psychology - EDUC PSYCHOL-UK, 1995; 15: 181-198.

13. Fraenkel, JR., Wallen N. E. How to design and evaluate research in education. New York, NY: McGraw-Hill. 2003.

14. Sawilowsky, D. Kelley L, Blair C, Markman BS. Meta-Analysis and the Solomon Four-Group Design, The Journal of Experimental Education, 1994; 62 (4): 361-376.

15. Felix A. Pre/Post-Testing to Evaluate the Effectiveness of Online Language Programs. JLTR. 2016; 4 (1): 176.

16. Delucchi M. Measuring Student Learning in Social Statistics: A Pretest-Posttest Study of Knowledge Gain. Teaching Sociology, 2014; 42 (3): 231–239.

17. Wood T, Cole G. Developing Multiple Choice Questions for the Royal College of Physicians and Surgeons of Canada Certification Examinations. Educational Research and Development, June, 2004: p. 4.

18. Chabot, Mira Costa, Chaffey, and Cabrillo Colleges. Boston University, SLO websites 12/17/2013; Revised 4/23/2014.

19. Martyn Shuttleworth. Pretest-Posttest Designs. Retrieved Nov 08, 2018 from Explorable.com: https://explorable.com/pretest-posttest-designs

20. Gibson KA, Boyle P, Black DA, Cunningham M, Grimm MC, McNeil HP. Enhancing evaluation in an undergraduate medical education program. Acad Med. 2008 Aug; 83 (8): 787–793.

21. Bhanji F, Gottesman R, de Grave W, Steinert Y, Winer LR. The retrospective pre-post: a practical method to evaluate learning from an educational program. Acad Emerg Med. 2012; 19 (2): 189-94.